﻿<!--
  Copyright 2013
  Ubiquitous Knowledge Processing (UKP) Lab
  Technische Universität Darmstadt
  
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at
  
    http://www.apache.org/licenses/LICENSE-2.0
  
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<html
	xmlns:wicket="http://wicket.apache.org/dtds.data/wicket-xhtml1.4-strict.dtd">
<head>
<link rel="stylesheet" href="css/style.css" type="text/css"></link>
</head>
<body>
	<div class="ui-state-highlight ui-corner-all infomargin">
		<div class="content">
			<h4>Quality control settings</h4>
			<ul>
				<li><b>Participation threshold</b>: percentage of users who have to
					annotate an item as <i>Correct</i> or <i>Wrong</i> for it to be
					considered <i>Correct</i>, <i>Wrong</i> or <i>Disputed</i>. If less
					than the given percentage of users have voted either <i>Correct</i>
					and <i>Wrong</i>, an item is considered <i>Incomplete</i>.</li>
				<li><b>Confidence threshold</b>: percentage of annotations the
					majority vote has to have over the minority. If it is set to 0,
					only items with a tie between <i>Correct</i> and <i>Wrong</i> votes
					are considered <i>Disputed</i>. If this is set to 1, any item which
					has not been voted for unanimously, is considered <i>Disputed</i>.
					A value of 0.5 indicates that an item is <i>Disputed</i> if one
					class (<i>Correct</i> or <i>Wrong</i>) has not at least twice as
					many votes as the other.<br /> Confidence is computed as
					abs(correct-wrong)/max(correct,wrong).</li>
			</ul>
			Consider the following example:<br /><br /> 10 testers, with 5 voting <i>Correct</i>
			and 3 voting <i>Wrong</i>, 2 having not voted.<br />
			The user ratio is (5+3)/10 = 0.8, and the confidence ratio is (5-3)/max(5,3) = 2/5 = 0.4.
			<br /><br /> We will show 3 scenarios:<br />
			<ol style="list-style-type: lower-alpha">
				<li>Participation threshold = 0.3, Confidence threshold = 0.6.<br />
					The item is labeled as <i>Disputed</i>, because the user ratio is high enough
					[0.8 &gt; 0.3] but the confidence ratio is not [0.4 &lt; 0.6].<br /><br />
				</li>
				<li>Participation threshold = 0.9, Confidence threshold = 0.4.<br />
					The item is labeled as <i>Incomplete</i>, because the user ratio is not high
					enough [0.8 &lt; 0.9].<br /><br />
				</li>
				<li>Participation threshold = 0.6, Confidence threshold = 0.3<br /> The
					The item is labeled as <i>Correct</i>, because the user ratio is high enough
					[0.8 &gt; 0.6], the confidence ratio is high enough [0.4 &gt; 0.3], and the
					there are more <i>Correct</i> than <i>Wrong</i> votes.<br /><br />
					<img src="images/statistics-example-a.png" alt="example" />
				</li>
			</ol>
		</div>
	</div>
</body>
</html>
